Resolve binlog MCP from the dnceng dotnet-public feed by YuliiaKovalova · Pull Request #969 · dotnet/skills

YuliiaKovalova · 2026-07-30T16:21:56Z

Summary

The binlog MCP server was declared as dnx Microsoft.AITools.BinlogMcp --yes with no feed pinned, so Microsoft.AITools.BinlogMcp resolved through whatever ambient NuGet configuration the agent happened to run under (user-level NuGet.Config, an enclosing repo's NuGet.config, etc.). That makes startup environment-dependent and hard to reproduce.

This pins the server to the dnceng dotnet-public feed via --source:

"args": [
  "dnx",
  "Microsoft.AITools.BinlogMcp",
  "--yes",
  "--source",
  "https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json"
]

--source (rather than --add-source) replaces all sources, so resolution is fully deterministic. This is safe here because .NET tool packages bundle their dependencies — only the single package needs to be restored.

The existing argument order is preserved, so the change is a pure insertion with no reordering.

Files changed

The server definition is duplicated across four manifests, all updated together so the mirrors stay in sync:

plugins/dotnet-msbuild/plugin.json
plugins/dotnet-msbuild/.claude-plugin/plugin.json
plugins/dotnet-msbuild/.codex-plugin/.mcp.json
.claude-plugin/marketplace.json

Two stale feed names are also corrected:

eng/allowed-external-deps.txt — comment said dotnet-eng feed
plugins/dotnet-msbuild/skills/binlog-failure-analysis/SKILL.md — fallback note said dotnet-tools NuGet feed

plugins/dotnet-msbuild/.codex-plugin/plugin.json needs no change — it delegates via "mcpServers": "./.mcp.json".

Validation

Package availability confirmed on dotnet-public: versions 1.0.0, 2.0.0, 2.0.1 (all stable, so --prerelease is not needed).

End-to-end server start against the pinned feed, piping an MCP initialize request into the server:

Server (aitools-binlog-mcp 2.0.1) method 'initialize' request handler called.
Server (aitools-binlog-mcp 2.0.1), Client (probe 1.0) method 'initialize' request handler completed in 20.5649ms.

CI gate reproduced locally — skill-validator check over all 16 plugins with --allowed-external-deps and --known-domains exits 0: 96 skills, 16 agents, 208 files scanned, 0 reference errors and no new warnings.
No known-domains.txt entry is required: ReferenceScanner only scans SKILL.md, *.agent.md, and references/*.md — not plugin.json.

Note on the failing `eval-quality` check

The check / Check eval quality job failure is pre-existing on main and unrelated to this PR — it fails on 4 stale tests/dotnet-maui/* entries in eng/eval-quality/underpowered-allowlist.txt that now meet the 5-trial floor:

eng/eval-quality/underpowered-allowlist.txt lists 'tests/dotnet-maui/maui-collectionview/eval.yaml',
but it now meets the 5-trial floor. Remove the line so the exemption can't be silently reused.

Same failure on main: run 30532730249. This PR touches no files under tests/ or eng/eval-quality/.

The `binlog` server ran `dnx Microsoft.AITools.BinlogMcp --yes` with no feed pinned, so resolution depended on whatever ambient NuGet configuration the agent happened to run under. Pin it to the dnceng dotnet-public feed with `--source` so the server resolves deterministically. Applied to all four manifests that embed the server definition (plugin.json, .claude-plugin/plugin.json, .codex-plugin/.mcp.json, and .claude-plugin/marketplace.json) so the mirrors stay in sync. Also corrects two stale feed names in comments/docs (dotnet-eng, dotnet-tools). Verified: `skill-validator check` passes (exit 0, no new warnings), and `dnx --yes --source <dotnet-public> Microsoft.AITools.BinlogMcp` starts aitools-binlog-mcp 2.0.1 and completes an MCP initialize handshake. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eef3a5cc-7c9b-41b2-88e5-fbc7e786f3f2

github-actions · 2026-07-30T16:22:34Z

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
✅	`dotnet-msbuild`	`check-bin-obj-clash`	4/5	80%
❌	`dotnet-msbuild`	`directory-build-organization`	0/1	0%
❌	`dotnet-msbuild`	`extension-points`	0/1	0%
✅	`dotnet-msbuild`	`msbuild-modernization`	6/7	85.7%
⚠️	`dotnet-msbuild`	`msbuild-server`	7/9	77.8%
❌	`dotnet-msbuild`	`property-patterns`	0/1	0%
✅	`dotnet-msbuild`	`resolve-project-references`	5/6	83.3%

Uncovered: dotnet-msbuild/check-bin-obj-clash

[WorkflowStep] Step 2: Get an overview and list projects (line 48)

Uncovered: dotnet-msbuild/directory-build-organization

[CodePattern] [MSBuild] (line 122)

Uncovered: dotnet-msbuild/extension-points

[CodePattern] [MSBuild] (line 195)

Uncovered: dotnet-msbuild/msbuild-modernization

[WorkflowStep] Step 6: Remove Unnecessary Boilerplate (line 217)

Uncovered: dotnet-msbuild/msbuild-server

[Validation] MSBUILDUSESERVER=1 is set in the shell (line 59)
[Validation] Second sequential build is faster than the first (line 60)

Uncovered: dotnet-msbuild/property-patterns

[CodePattern] [MSBuild] (line 33)

Uncovered: dotnet-msbuild/resolve-project-references

[Validation] ResolveProjectReferences was not set as the optimization target (line 64)

Copilot

Pull request overview

This PR aims to make the binlog MCP server startup deterministic by pinning Microsoft.AITools.BinlogMcp restore to the dnceng dotnet-public NuGet feed, avoiding dependence on ambient NuGet configuration on the host environment.

Changes:

Updated the binlog MCP server invocation to include a pinned NuGet source (--source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json) across the mirrored manifests.
Corrected stale feed wording in documentation and the allowed-external-deps comment to reference dotnet-public.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
plugins/dotnet-msbuild/skills/binlog-failure-analysis/SKILL.md	Updates fallback documentation to reference the correct feed name (`dotnet-public`).
plugins/dotnet-msbuild/plugin.json	Adds a pinned NuGet feed to the binlog MCP server definition.
plugins/dotnet-msbuild/.codex-plugin/.mcp.json	Mirrors the pinned-feed binlog MCP server definition for Codex plugin consumption.
plugins/dotnet-msbuild/.claude-plugin/plugin.json	Mirrors the pinned-feed binlog MCP server definition for Claude plugin consumption.
eng/allowed-external-deps.txt	Updates the explanatory comment to reflect the dotnet-public feed.
.claude-plugin/marketplace.json	Mirrors the pinned-feed binlog MCP server definition in the marketplace manifest.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Addresses review feedback. Both orderings resolve and start the server correctly (verified), but keeping `Microsoft.AITools.BinlogMcp` immediately after `dnx` matches the ordering already used in this repo and turns the change into a pure insertion of `--source` with no reordering noise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eef3a5cc-7c9b-41b2-88e5-fbc7e786f3f2

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

YuliiaKovalova · 2026-07-30T16:36:55Z

/evaluate d70cc01

github-actions · 2026-07-30T16:45:27Z

📊 Skill Evaluation Results

1 skill(s) evaluated — ✅ 0 improved, ❌ 0 no credible change, 🔻 0 regressed.

⚠️ 1 could not be judged: 1 underpowered — the eval has fewer trials than any result needs to reach p ≤ 0.05, so no verdict was possible. This is the eval's size, not a skill regression; fix it by adding scenarios or raising defaults.runs.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
binlog-failure-analysis	⚠️	+100.0%	0.500	+100.0%	1/0/0	5.0/5	0.0/5	✅ 0.06	1/1 · 1/1 (plugin)

ℹ️ Column legend

Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
W/T/L — wins / ties / losses across trials.
⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

⚠️ binlog-failure-analysis — details

Reason: Net win +100.0% (1W/0T/0L over 1 trial(s), sign test p=0.500), mean preference +100.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5, and this eval won every one of them) — raise the eval's trial count with more scenarios or defaults.runs

Scenario	Net win	Δ Pref	Trials (W/T/L)
▲ Diagnose build failures from binlog only (no source files)	+100.0%	+100.0%	1/0/0

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

ViktorHofer

This looks questionable to me. We should never promote a package to customers from the dotnet-public nuget feed. That's a public feed but exists only for our own engineering infrastructure (to build .NET). cc @mmitche to double check my statement

github-actions · 2026-07-30T17:50:51Z

👋 @YuliiaKovalova — this PR has changes requested. When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

YuliiaKovalova · 2026-07-31T07:51:17Z

This looks questionable to me. We should never promote a package to customers from the dotnet-public nuget feed. That's a public feed but exists only for our own engineering infrastructure (to build .NET). cc @mmitche to double check my statement

I had conversation about feeds here befroe dotnet/dotnet-buildtools-prereqs-docker#1682 (comment)

these cases look similar to me, but i might be wrong.

github-actions · 2026-08-03T17:58:21Z

👋 @YuliiaKovalova — this PR has changes requested. When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

Evangelink · 2026-08-03T18:44:40Z

/evaluate

github-actions · 2026-08-03T18:44:52Z

👋 /evaluate needs the exact commit to evaluate.

Two ways to run it:

Review flow (recommended — no SHA to copy): open Files changed → Review changes, type /evaluate in the review box, and Submit review. GitHub binds the run to the exact commit you reviewed.
Comment flow: copy-paste this command for the current head:

/evaluate 73e8ad89848d842de30bccc9aec1e0b5dd71419d

Copilot

Copilot's findings

Files reviewed: 6/6 changed files
Comments generated: 0 new

got confirmation that change is correct

github-actions · 2026-08-04T09:23:13Z

📊 Skill Evaluation Results

1 skill(s) evaluated — ✅ 0 improved, ❌ 0 no credible change, 🔻 0 regressed.

⚠️ 1 could not be judged: 1 underpowered — the eval has fewer trials than any result needs to reach p ≤ 0.05, so no verdict was possible. This is the eval's size, not a skill regression; fix it by adding scenarios or raising defaults.runs.

A skill passes only on a credible net win over baseline: more wins than losses, by an exact one-sided sign test at p ≤ 0.05.

Skill	Result	Net win	p	Δ Pref	W/T/L	Quality	Baseline	Overfit	Skills Loaded
binlog-failure-analysis	⚠️	+0.0%	1.000	+0.0%	0/1/0	0.0/5	0.0/5	✅ 0.08	1/1 · 1/1 (plugin)

ℹ️ Column legend

Net win — (wins − losses) / trials for skilled vs baseline, judged head-to-head by vally compare. This is the effect the gate decides on.
p — one-sided exact sign test over the discordant (non-tie) trials. A skill passes only at p ≤ 0.05, which needs at least 5 winning trials.
Δ Pref — the same comparison weighted by how decisive each win was (much-better ±100%, slightly-better ±40%). Reported for triage only: weighting the statistic by magnitude made a skill fail for winning harder, which is why the gate deliberately ignores this column.
W/T/L — wins / ties / losses across trials.
⚠️ — the gate withheld a verdict. Either the eval has fewer trials than any result needs to reach p ≤ 0.05 (underpowered — the skill was never actually measured, so this is not a regression; add scenarios or raise defaults.runs), or the comparison didn't complete.
🔻 — a credible regression: the losses themselves clear the same bar the gate uses for wins.
Quality / Baseline — mean absolute judge score 0–5 (skilled isolated vs skill-free control).
Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) with its score.
Skills Loaded — of the scenarios that expect activation, how many actually activated / that total (plugin run shown when present); ⚠️ marks a scenario that expected activation but didn't activate.

⚠️ binlog-failure-analysis — details

Reason: Net win +0.0% (0W/1T/0L over 1 trial(s), sign test p=1.000), mean preference +0.0% — underpowered (1 counted trial(s); a credible verdict needs at least 5) — raise the eval's trial count with more scenarios or defaults.runs

Scenario	Net win	Δ Pref	Trials (W/T/L)
= Diagnose build failures from binlog only (no source files)	+0.0%	+0.0%	0/1/0

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

ViktorHofer · 2026-08-04T09:24:24Z

got confirmation that change is correct

Apparently requesting changes doesn't matter anymore? Next time share context before dismissing a change request or even better, don't dismiss at all. That doesn't feel right.

YuliiaKovalova · 2026-08-04T09:40:41Z

got confirmation that change is correct

Apparently requesting changes doesn't matter anymore? Next time share context before dismissing a change request or even better, don't dismiss at all. That doesn't feel right.

I apologize, it looked to me like I had enough evidence to proceed.
If you have technical concerns I can revert this change and start the discussion from scratch.

ViktorHofer · 2026-08-04T10:20:43Z

Thanks for being considerate, really appreciate that. Let's move the discussion offline. I opened a Teams channel to collaborate further on the dotnet-msbuild plugin.

JanKrivanek · 2026-08-04T10:34:58Z

@YuliiaKovalova - were there any cases where the tooling didn't work as expected?

I'd prefer leaving the feed definition up to destination env - similar to e.g. Roslyn lsp https://github.com/dotnet/skills/blob/main/plugins/dotnet/lsp.json

Copilot AI review requested due to automatic review settings July 30, 2026 16:21

YuliiaKovalova requested review from JanKrivanek and ViktorHofer as code owners July 30, 2026 16:21

YuliiaKovalova requested a review from a team July 30, 2026 16:21

YuliiaKovalova requested review from AbhitejJohn and webreidi as code owners July 30, 2026 16:21

Copilot started reviewing on behalf of YuliiaKovalova July 30, 2026 16:22 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

Comment thread plugins/dotnet-msbuild/plugin.json Outdated

Comment thread plugins/dotnet-msbuild/.claude-plugin/plugin.json Outdated

Comment thread plugins/dotnet-msbuild/.codex-plugin/.mcp.json Outdated

Comment thread .claude-plugin/marketplace.json Outdated

Evangelink approved these changes Jul 30, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 30, 2026 16:33

Copilot started reviewing on behalf of YuliiaKovalova July 30, 2026 16:33 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

YuliiaKovalova mentioned this pull request Jul 30, 2026

Resolve binlog MCP from the dnceng dotnet-public feed dotnet/arcade-skills#49

Merged

ViktorHofer previously requested changes Jul 30, 2026

View reviewed changes

github-actions Bot added the waiting-on-author PR state label label Jul 30, 2026

mmitche approved these changes Aug 3, 2026

View reviewed changes

AbhitejJohn approved these changes Aug 3, 2026

View reviewed changes

Merge branch 'main' into binlog-mcp-dotnet-public-feed

73e8ad8

Copilot AI review requested due to automatic review settings August 3, 2026 18:43

Evangelink enabled auto-merge (squash) August 3, 2026 18:43

Copilot started reviewing on behalf of Evangelink August 3, 2026 18:44 View session

Evangelink requested a review from ViktorHofer August 3, 2026 18:44

Copilot AI reviewed Aug 3, 2026

View reviewed changes

github-actions Bot added pr-state/ready-for-eval PR is mergeable and awaiting evaluation and removed waiting-on-author PR state label labels Aug 4, 2026

Evangelink merged commit 805a42a into main Aug 4, 2026
35 checks passed

Evangelink deleted the binlog-mcp-dotnet-public-feed branch August 4, 2026 09:20

Evangelink mentioned this pull request Aug 4, 2026

Make BinlogMcp feed-neutral and configure eval NuGet source #984

Merged

AbhitejJohn mentioned this pull request Aug 4, 2026

Move eight P0/P1 skills to staging #972

Open

opencode-agent Bot mentioned this pull request Aug 4, 2026

Zero conflicts. 16 plugins, 96 skills. elvisw/dotnet-skills#30

Open

Uh oh!

Conversation

YuliiaKovalova commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Files changed

Validation

Note on the failing eval-quality check

Related

Uh oh!

github-actions Bot commented Jul 30, 2026

Skill Coverage Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

YuliiaKovalova commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

📊 Skill Evaluation Results

Uh oh!

ViktorHofer left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

YuliiaKovalova commented Jul 31, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

Evangelink commented Aug 3, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot's findings

Uh oh!

Uh oh!

github-actions Bot commented Aug 4, 2026

📊 Skill Evaluation Results

Uh oh!

ViktorHofer commented Aug 4, 2026

Uh oh!

YuliiaKovalova commented Aug 4, 2026

Uh oh!

ViktorHofer commented Aug 4, 2026

Uh oh!

JanKrivanek commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

YuliiaKovalova commented Jul 30, 2026 •

edited

Loading

Note on the failing `eval-quality` check